Method and system for preparing video communication image for wide screen display

ABSTRACT

A system and method is disclosed for composing a video layout having an aspect ratio of 16:9 from a plurality of video segments having an aspect ratio other than 16:9. For example, the plurality of video segments might have an aspect ratio of 4:3. To create a composite frame having an aspect ratio of about 16:9, the disclosed system scales each one of the segments and places each scaled segment in the layout in such a way that the composite layout has an aspect ratio of about 16:9.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/851,448, filed May 21, 2004, now U.S. Pat. No. 7,113,200, thecontents of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the field of video communication andmore particularly to a method and system of preparing images for widescreen displays in video communication.

2. Description of the Related Art

Recently, a new type of video display, known as a Wide Screen display,has been introduced into the market. As is known, the aspect ratio of avideo display or screen is the ratio of the width of the picture to theheight of the picture. Common video screens have an aspect ratio ofabout 4:3. In contrast, a wide screen display has an aspect ratio ofabout 16:9, which is about the same aspect ratio of cinema screens. Formost current TV sets, the aspect ratio is about 4:3. For a highdefinition TV (HDTV), the ratio is about 16:9. An exemplary resolutionof a wide screen display may be about 1024×576 pixels (width×height),while the resolution of a common video display is about 640×480 pixels.Common resolution for video communication is Common Intermediate Format(CIF), which is about 352×288 pixels. To match the aspect ratio of avideo display of about 4:3, the aspect ratio of a pixel for CIF is about12:11. The 12:11 aspect ratio for each pixel creates an image with anentire aspect ratio of (352*12) to (288*11), which is about (4224:3168)or (4:3).

Common video communication equipment, which includes but is not limitedto a Video Communication Control Unit (VCCU) or endpoints, normally usesvideo images with resolutions such as Quarter of Common IntermediateFormat (QCIF), CIF, or 4CIF (four times Common Intermediate Format) withan aspect ratio of 4:3. An exemplary VCCU may be a Multipoint ControlUnit (MCU), a Multimedia Gateway, etc. Typically, a VCCU serves as aswitchboard and/or conference builder for the network. In operation, theVCCU receives and transmits coded video streams to and from various userterminals or codecs. A Multipoint Control Unit (MCU) is conferencecontrolling equipment typically located in a node of a network or in aterminal. The MCU receives several channels from access ports, processesaudio and visual signals according to certain criteria, and distributesthe processed signals to a set of connected channels. An example of anMCU includes the MGC-100, which is available from Polycom Inc., theassignee of the present disclosure. The MCU is a video communicationcontroller that can be used in other types of video communication. Itshould be noted that the terms “MCU” and “VCCU” may be usedinterchangeably in the present disclosure.

An endpoint or a terminal is an entity on the network and is capable ofproviding real-time, two-way audio, visual, and/or data communicationwith other terminals or with the MCU. Currently, if an endpoint isassociated with a wide screen display instead of a common 4:3 screendisplay, the endpoint as well as the other endpoints and the VCCU thatare involved in the same video conference are not aware of the aspectratio of 16:9 for the wide screen display. Therefore, the video imagethat the wide screen display receives from its associated endpoint has acommon aspect ratio of 4:3. Typically, a wide screen display has threemain types of settings, and a user may set the wide screen display usingthe control panel on the wide screen display to one of these settings.Currently, an image with an aspect ratio of 4:3 may be displayed on awide screen display using one of the settings described below inconjunction with FIGS. 1A-1C.

Referring to FIG. 1A, a wide screen display 110 set to a normal settingshows a video image 112. In the normal setting, the video image 112 isdisplayed as is on the wide screen display 110. In other words, thevideo image 112 has the aspect ratio of 4:3, while the wide screendisplay has the aspect ratio of 16:9. Therefore, portions of the widescreen display 110 are not used, as illustrated by the black sections114 in FIG. 1A.

Referring to FIG. 1B, a wide screen display 110 set to a zoom settinghas a video image 120. In the zoom setting, the video image 120 issymmetrically enlarged by the internal electronics of the wide screendisplay 110. Therefore, portion of the image, in the top and/or thebottom, is cropped. Unfortunately, the cropped area can be an importantpart of the image, such as the face of a participant.

Referring to FIG. 1C, a wide screen display 110 set to a wide settinghas a video image 130. In the wide setting, the electronics of the widescreen display 110 enlarges the regular image by asymmetrical factors sothat the enlarged image substantially fits the wide screen display 110.To convert the regular aspect ratio of 4:3 to the wide screen aspectratio of 16:9, the width of the regular image is increased by a factorof four, and the height is increased by a factor of three. The result isa distorted image 130, as shown in FIG. 1B. For example, a circlebecomes elliptical, and a person becomes stretched wider. It isunderstood that wide screen displays may use other terminology for the‘normal, ‘zoom’ and ‘wide’ settings. In addition, other wide screendisplays may have additional settings, which can be a combination of oneor more of the above settings.

Current techniques for connecting a wide screen display to a videocommunication, as described above in conjunction with FIGS. 1A-1C, areinsufficient. Therefore, a need exists in the art for a system andmethod for connecting a wide screen display to a videoconference. Thesubject matter of the present disclosure is directed to overcoming, orat least reducing the effects of, one or more of the problems set forthabove.

SUMMARY OF THE PRESENT DISCLOSURE

The system and method of the present disclosure overcomes theabove-described need in the prior art by providing techniques for usinga wide screen display in a video conference. The disclosed system andmethod can inform the VCCU or MCU about the type of screen connected toeach one of the endpoints participating in the video conference.Informing the VCCU or MCU can be done prior to starting the videoconference or when establishing the video conference.

During the video conference, uncompressed video image data is encodedand transmitted to an endpoint with a wide screen display. The videoimage is processed before being encoded (compressed) so that the imagecan be displayed properly on the wide screen display. In the presentdisclosure, uncompressed video images refer to video data in a spatialdomain (image domain) or in a transform domain (i.e. a DCT domain). Itshould be noted that the terms “uncompressed video,” “open video,” and“decoded video” may be used interchangeably in the present disclosure.

Processing the uncompressed video image at the VCCU can involve twostages. In a first stage, a composite video image having the aspectratio of 16:9 is prepared by creating a layout from two or more videoimages having the aspect ratio of 4:3. In a second stage, the 16:9composite video image is processed to fit the capabilities of theendpoint to handle an image with an aspect ratio of 16:9. An endpointassociated with a wide screen display can handle video images with anaspect ratio of 16:9 if the endpoint uses an H.264 or an H.263 with‘Custom Picture Formats’ capability, for example.

However, endpoints that can handle only an H.261 or an H.263 without‘Custom Picture Format’ capability cannot process an image having anaspect ratio of 16:9. Hence, these endpoints can only handle imageshaving aspect ratio of 4:3. Therefore, the uncompressed composite videoimage with the 16:9 ratio has to be manipulated or altered before beingtransmitted to these endpoints. Once the composite video image ismanipulated or altered, the endpoint receives and decodes the alteredcompressed video image. Then, the endpoint delivers the uncompressedvideo image to its associated wide screen display. The electronics ofthe wide screen display then reverse the manipulation or alteration ofthe uncompressed video image. Consequently, the wide screen displaysubstantially displays the video image with an aspect ratio of 16:9without the typical forms of distortion noted above with reference toFIGS. 1A-1C.

The disclosed system and method preferably solves the problem of how toinform a VCCU about the type of the screen that is associated with theendpoints. Current video communication protocols include but are notlimited to H.320; H.324; H.323, etc. The current video communicationprotocols do not define any control or signaling that can be used todefine the type of the screen at the endpoint. Informing the VCCU aboutthe type of screen associated with an endpoint can be done by differenttechniques.

In one embodiment for informing the VCCU about the type of screenassociated with an endpoint, the endpoint can inform the VCCU about thetype of the screen when reserving the video conference. The aspect ratioor the type of the screen can be added as one of the parameters that arerequired during reserving a video conference. In other embodiments forinforming the VCCU about the type of screen associated with an endpoint,an interactive voice response (IVR) session can be done whenestablishing the video conference or during the conference. Aparticipant can be asked to select the type of the screen that is usedfrom a variety of options. The participant can respond to the questionsby pressing an appropriate key on a keypad at the endpoint and sendingsignals to the VCCU. The signals can include but are not limited to DualTone Modulation Frequency (DTMF) signals or Far End Camera Control(FECC) signals. The keypad can be the keypad of the remote control ofthe endpoint, the dialing key of a phone, etc. Other embodiments forinforming the VCCU about the type of screen associated with an endpointcan use other signaling techniques, such as speech recognition. In stillother embodiments for informing the VCCU, the disclosed system andmethod can use techniques for controlling multimedia video communicationdisclosed in U.S. patent application Publication No. 20030174202, whichhas Ser. No. 10/346,306 and is incorporated herein by reference in itsentirety.

After receiving the type of screen associated with the endpoints, theMCU prepares and delivers compressed and manipulated video images tothose endpoints with wide screens. The compressed and manipulated videoimages match one of the common compressed algorithms that can be used bythe endpoint. Some common compression algorithms include but are notlimited to H.261, H.263, H.264, and MPEG. The compressed video image ismanipulated or altered in such a way that it can be processed by theendpoint. When the manipulated video image is displayed on the widescreen, it can be manipulated by the wide screen to substantially coverthe entire screen without distortions.

For an endpoint that cannot use H.264 or H.263 with ‘Custom PictureFormat’ capability as the compression algorithm, the aspect ratio thatcan be received and processed by the endpoint is about 4:3. Therefore,the disclosed system and method generates an uncompressed video imagehaving the aspect ratio of 16:9, manipulates the image to reach thedesired aspect ratio of 4:3, encodes the image, and sends the image tothe endpoint.

In one embodiment of manipulating or altering the video image, thedisclosed system and method generates an uncompressed video image, whichcan be a composite layout of two or more images. The uncompressed videoimage has the aspect ratio of 16:9. It should be noted that the terms“composite layout”, “composite frame” and “composite image” may be usedinterchangeably throughout the present disclosure. Then, one or tworectangular segments having a background color (e.g., no information) isadded on the top and/or the bottom of the 16:9 video image. Theadditional one or two segments change the aspect ratio of themanipulated image to 4:3, which can be processed according to H.261 orH.263 without ‘Custom Picture Format’ capability. The total height ofthe additional segments can be one third of the height of the composite16:9 image. The manipulated, uncompressed video is encoded andtransmitted to the endpoint. At the endpoint, the 4:3 compressed imageis decoded and transferred to the wide screen display. The wide screendisplay set by the user to zoom mode symmetrically enlarges themanipulated image to substantially cover the width of the wide screendisplay. The additional segments with the background color are cropped,and a full screen of the composite image with an aspect ratio of 16:9 issubstantially displayed on the wide screen display.

In another embodiment of manipulating or altering the video image, thedisclosed system and method generates an uncompressed video image havingthe aspect ratio of 16:9. Then, the video image is distorted by ascaler. The scaler increases the resolution of a height of the image bya factor of four thirds, creating a manipulated uncompressed videohaving the aspect ratio of 4:3. The manipulated image is encoded andtransmitted to the endpoint. The endpoint decodes the manipulated imageand delivers it to the wide screen display. The wide screen display isset to the wide setting in which the manipulated image is asymmetricallyenlarged by four thirds. This asymmetric enlargement corrects theprevious distortion, which was done prior to encoding the video image.As a result, the wide screen display substantially displays the imagewithout distortion and with the aspect ratio of 16:9 on the wide screen.

The foregoing summary is not intended to summarize each potentialembodiment or every aspect of the present disclosure. Other features andadvantages of the present invention will become apparent upon readingthe following detailed description of the embodiments with theaccompanying drawings and appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, exemplary embodiments, and other aspects ofsubject matter of the present disclosure will be best understood withreference to a detailed description of specific embodiments, whichfollows, when read in conjunction with the accompanying drawings, inwhich:

FIGS. 1A-1C illustrate examples of how a video image having an aspectratio of 4:3 is displayed on a wide screen display using differentsettings;

FIG. 2 is a block diagram illustrating an exemplary embodiment of avideo unit within a VCCU;

FIGS. 3A-3F illustrate exemplary layouts that may be used with a widescreen display;

FIGS. 4A-4C illustrate an exemplary video image in three differentstages of processing to be displayed over a wide screen display set to azoom mode;

FIGS. 5A-5C illustrate an exemplary video image in three differentstages of processing to be displayed over a wide screen display set to awide mode; and

FIG. 6 is a flowchart showing an exemplary method for setting a videounit for preparing compressed video stream to be sent and displayed overa wide screen display.

While the disclosed system and method are susceptible to variousmodifications and alternative forms, specific embodiments thereof havebeen shown by way of example in the drawings and are herein described indetail. The figures and written description are not intended to limitthe scope of the inventive concepts in any manner. Rather, the figuresand written description are provided to illustrate the inventiveconcepts to a person skilled in the art by reference to particularembodiments, as required by 35 U.S.C. §112.

DETAILED DESCRIPTION

Referring now to the drawings, in which like numerals refer to likeparts throughout the several views, exemplary embodiments of the presentdisclosure are described.

FIG. 2 is a block diagram showing some of the logical elements of anexemplary video unit 200 within a VCCU. The video unit 200 processes andprepares compressed video images for transmission to an endpoint (notshown) having a wide screen display (not shown). In the presentembodiment, the video unit 200 includes a compressed video commoninterface (CVCI) 205; a plurality of input modules 210 a-c; a pluralityof output modules 220 a-c; and a Decoded Video Common Interface (DVCI)230. Each input module 210 a-c includes an input buffer 212, a decoder214, and one or more input scalers 216 a-c. Each output module 220 a-cincludes a builder 222; an output scaler 224; an encoder 226; and anoutput buffer 228. Three units of input modules 210 a-c, scaler 216 a-cand output module 220 a-c are shown in FIG. 2 by way of example. It isunderstood that the disclosed system and method can be used with anynumber of modules. In addition, it is understood that other video unitswithin a VCCU may have a different configuration than that schematicallyshown and described here.

A plurality of endpoints or terminals (not shown) is connected over oneor more communication networks (not shown) to the VCCU. It should benoted that the terms “endpoints” and “terminals” may be usedinterchangeably herein. The endpoints send their compressed videostreams to the appropriate input modules 210 a-c via Network InterfaceModules (not shown) and the CVCI 205. Also, the endpoints receive thecompressed video streams from the appropriate output modules 220 a-c viaNetwork Interface Modules (not shown) and the CVCI 205. Thecommunication networks can be packet-based networks, circuit switchednetworks, and/or other networks or communication protocols, such asISDN, ATM, PSTN, cellular and/or IP. The disclosed system and method isnot limited to a particular type of communication protocol or type ofnetwork.

The CVCI 205 routes the compressed video streams between the inputmodules 210 a-c, the output modules 220 a-c, and the network interfacemodules (not shown). The CVCI 205 can be a TDM bus, packet based bus(such as an ATM bus, IP bus), serial bus, parallel bus, connectionswitching, shared memory, direct connection, or any variety of these.The operation of the video unit 200 is controlled by a central controlunit (not shown). The central control unit can be a host computer orinternal module of the VCCU. In the present disclosure, the centralcontrol unit is referred to as a management conference system (MCS).

Each input module 210 a-c and/or output module 220 a-c can be a logicalunit, a hardware module, a firmware module, a software module, or anycombination of these. Each module 210 a-c, 220 a-c can be a permanentlogical module or a temporary one, which is generated by the MCSaccording to current needs. Generating temporary logical modules andallocating permanent modules according to the current needs conservesthe resources of the VCCU.

The number of the input modules 210 a-c that may be associated with avideo conference can be a fixed number or can be varied according to theneeds of the video conference. For example, a video conference may needone input module 210 for each of the endpoints participating in thevideo conference. In another video conference, one input module 210 canbe used for each currently visible participant in the relevant screenlayout.

Each video conference can be associated with one or more output modules220 a-c. In an exemplary embodiment of the disclosed system, one outputmodule 220 is used for each of the endpoints participating in the videoconference. In an alternative embodiment, one output module 220 can beused for each type of screen layout, and each output module 220 cantransfer its output to the endpoints that are using the particular typeof layout. Allocation of the output modules 220 can depend on variousparameters known in the art, including but not limited to bit rate,frame rate, and coding algorithm, for example.

Once a compressed input video stream from an endpoint associated with acertain input module 210 is placed onto the CVCI 205, the input videostream begins to accumulate in the input buffer 212. The accumulation inthe buffer 212 is done according to the type of the CVCI 205. Forexample, if the CVCI 205 is a TDM bus, then the buffer 212 grabs theappropriate input stream by sampling the CVCI 205 at the time slotassigned to the endpoint associated with the input module 210. However,the buffer 212 may not be necessary, and other embodiments of thedisclosed system may accumulate the appropriate compressed input streamby other techniques known in the art.

The decoder 214 takes the received compressed video stream from inputbuffer 212. Based on the encoding standards (H.261, H.263, H.264, etc.)of the compressed video stream, the decoder 214 decodes the compressedvideo stream into an uncompressed video, which can then be representedin the image (spatial) domain.

The output from the decoder 214 is transferred from the decoder 214 toone or more input scalers 216 a-c. The number of input scalers 216depends on the number of different segment sizes to which theuncompressed video is assigned in the different layouts of theconference. The scaling performed by the input scalers 216 a-c changesthe resolution of the uncompressed video according to the requirementsof the endpoints and/or according to the size of the associatedsegments. The input scalers 216 a-c can also perform known filteroperations on the scaled, uncompressed video to preserve the picturequality. The output of the input scalers 216 a-c is transferred to theDecoded Video Common Interface (DVCI) 230. The input scalers 216 a-c canbe connected to the DVCI 230 via a buffer (not shown), and the transferof the uncompressed video can performed according to the type of DVCIused for the disclosed system. For example, if the DVCI 230 is a TDMbus, then a decoded stream from an input scaler 216 a-c can betransferred to the DVCI 230 during a time slot associated with theparticular decoded stream.

In an alternative embodiment of the disclosed system, the input modules210 a-c may not include input scalers 216. Instead, the decoded streamfrom the decoder 214 can be transferred directly or via a buffer (notshown) to the DVCI 230. The builder 222 on the output module 220 a-c canthen perform the various functions of the input scalers 216.

In the present embodiment, the DVCI 230 routes the decoded video betweenthe input modules 210 a-c and the output modules 220 a-c. The DVCI 230can be a TDM bus, packet-based bus, serial bus, parallel bus, connectionswitching, shared memory, direct connection, or any variety of these. Inalternative embodiments, the disclosed system can use the CVCI 205 forthe same functions as the DVCI 230.

The builders 222 on the appropriate one or more output modules 220 a-cretrieve one or more scaled decoded video (uncompressed video) segmentsfrom the DVCI 230. The selection of the decoded video segments is basedon the layout needed to be generated by the output module 220. Thebuilder 222 collects decoded video segments from one or more inputmodule 210 a-c as needed by the layout. The builder 222 then arrangesthe decoded video segments according to the layout of the videoconference to create a composite frame. In other embodiments of thedisclosed system, the builder 222 can scale the received decoded frameto fit the size of its associated segment in the layout of the videoconference. The aspect ratio of the output frame at the output ofbuilder 222 can be about 4:3 or 16:9 depending on the aspect ratio ofthe display used by the endpoint associated with the relevant outputmodule 220.

To create a composite frame having an aspect ratio of about 16:9 fromtwo or more segments that each have an aspect ratio of about 4:3, thedisclosed system scales each one of the segments and places each scaledsegment in the layout in such a way that the composite layout has anaspect ratio of about 16:9. Referring to FIGS. 3A-3D, embodiments ofexemplary layouts 320, 330, 340 and 350 for creating a composite framefrom scaled segments are schematically illustrated. In each of theexemplary layouts 320, 330, 340, and 350, a composite frame 322, 332,342, and 352 with an aspect ratio of about 16:9 is divided into aninteger number of segments having the aspect ratio of about 4:3. Forexample, the layout 320 of FIG. 3A has a composite frame 322 composed oftwelve segments (320 a to 320L) with each segment having an aspect ratioof about 4:3 and the segments arranged in a 4×3 matrix.

The layout 320 of FIG. 3A can be used as a basic layout from whichadditional layouts with integer numbers of 4:3 segments can be composedto generate an overall layout of about 16:9. For example, the otherlayouts 330, 340 and 350 in FIGS. 3B-3D illustrate derivatives of layout320 and have different combinations of the basic division of twelveshown in the layout 320. These other layouts 330, 340 and 350 can beused in other conferences according to the needs of the videoconference.

Some video conferences may require layouts where an integer number ofsegments having aspect ratios of about 4:3 will not cover the entire16:9 display screen. For example, a video conference may require alayout having six segments that are equal in size. In one embodiment,the disclosed system can create a layout 360 a shown n FIG. 3E. Eachsegment 360 a to 360 f of the layout 360 a has an aspect ratio of about4:3, and the uncovered area 362 is filled with background.

In an alternative embodiment, the disclosed system can create a layout360b shown in FIG. 3F when a video conference requires a layout havingsix segments that are equal in size. The disclosed system scales each ofthe six segments 364 a to 364 f such that the total area covered by thesix segments is bigger than the 16:9 frame. Then, the edges of eachsegment may be cropped to reduce the total size to fit the frame. Inlayout 360 b, each segment 364 a to 364 f has been scaled and cropped.Since the cropped area is divided over the six segments 364 a to 364 fit is less disturbing than cropping portions of the entire frame as seenin the example of FIG. 1B. The above technique of scaling and croppingthe segments may be used if one or more inputs have an aspect ratioother than 4:3 (e.g., 16:9).

Returning to FIG. 2, one or more input scalers 216 a-c in someembodiments of the disclosed system can be adjusted to scale the decodedvideo generated by the decoder 214 to the appropriate one or more sizesof the image in different layouts. Other embodiments of the disclosedsystem can have one or more scalers integrated into the builder 222rather than having separate input scalers 216 a-c. The builder 222 canbe adjusted to place the appropriate decoded streams in their locationin the layouts.

After the builder 222 creates the composite frame of the video image,the output scaler 224 in the present embodiment then scales the videoimage to the desired resolution and transfers the scaled video image tothe encoder 226. If the endpoint associated with the output module 220has a wide screen display but the endpoint cannot process a video imagehaving an aspect ratio of 16:9, additional manipulation is required toconvert the video image from the aspect ratio of about 16:9 to 4:3before transferring the image to the encoder 226.

In one embodiment of manipulating or altering the video image discussedin more detail below with reference to FIGS. 4A-4C, the builder 222 addsa rectangular segment to the height of the composite image. The heightof this additional rectangular segment can be one third of the height ofthe composite image. This additional rectangular segment can be dividedinto two halves. One half can be placed above the composite image, andthe other can be placed below the composite image. The additionalrectangular segments can be filled with background color.

In another embodiment of manipulating or altering the video imagediscussed in more detail with reference to FIGS. 5A-5C, the outputscaler 224 distorts the image by enlarging an original height H of theimage by a factor of 4/3 compared to the width of the image. Thisscaling converts the aspect ratio from about 16:9 to about 4:3. As notedabove, some embodiments of the disclosed system may not require outputscalers 224 in which case one or more input scalers 216 a-c can distortthe decoded image instead of an output scaler

In the present embodiment, the encoder 226 receives the scaled compositeimage from the output scaler 224 or builder 222. The encoder 226 encodesthe scaled composite image according to the compression algorithm usedby the one or more endpoints associated with the encoder 226. Forexample, the compression algorithm can be H.261, H.263, H.264, etc.,which are known in the art and not discussed herein. The compressedvideo stream of the image is then transferred via the output buffer 228,the CVCI 205, and one or more network interface modules (not shown) tothe appropriate endpoints (not shown).

Other details of the video unit 200 can be found in U.S. Pat. Nos.5,600,646; 5,838,664; and 6,300,973, which are incorporated herein byreference in their entireties. Additional details can also be found inU.S. Patent Application Publication No. 20040042553, which has Ser. No.10/344,792 and is incorporated herein by reference in its entirety.

FIGS. 4A-4C illustrate an example video image during three differentprocessing stages of a composite image. FIG. 4C illustrates theoperation and the output over the wide screen in the endpoint. Becausethe endpoint associated with a wide screen display may not be capable ofreceiving 16:9 video images, the video image is manipulated or alteredto be displayed on the wide screen display set to the zoom mode. In thezoom mode, the image is enlarged symmetrically to be display oversubstantially the entire wide screen of the display.

The builder 222 (FIG. 2) composes an example composite image 410, asshown in FIG. 4A. The composite image 410 has four segments, whichinclude a large segment 411 and three small segments 414, 416 and 418.The large segment 411 can be associated with the current speaker in thevideo conference, and the other segments can be associated with the restof the participants of the video conference. The builder 222 (FIG. 2)collects the video data of the different segments 411, 414, 416, and 418from the decoded video common interface 230 (FIG. 2). The builder 222(FIG. 2) then composes one frame 410 with an aspect ratio of 16:9 usingthe different segments 411, 414, 416, and 418, where each one of thesegments 411, 414, 416, and 418 has an aspect ratio of 4:3.

Next, the builder 222 (FIG. 2) manipulates the composite video toproduce an image 420 of FIG. 4B where the aspect ratio of the entireimage is about 4:3. The manipulated image 420 can be transmitted to anendpoint using a wide screen display where the endpoint is capable ofhandling incoming streams having an aspect ratio of about 4:3. Toconvert the aspect ratio to 4:3, the builder 222 (FIG. 2) adds tworectangular segments 422 a and 422 b to form the manipulated compositeimage 420. Each of the additional segments 422A and 422B can be filledwith a background color. The total height of both of the additionalrectangular segments 422A and 422B is equal to about a third of theoriginal height H of the composite image 410. The composite image 410together with the two rectangular segments 422A and 422B create amanipulated and uncompressed video (decoded video) having the aspectratio of about 4:3.

The image 420 of FIG. 4B is transferred via the output scaler 224 (FIG.2) to the encoder 226 (FIG. 2). The image 420 is compressed according tothe compression standard used by the receiving endpoint (not shown) andis sent as a compressed video to the endpoint. At the endpoint, thecompressed video is decoded. The manipulated decoded video has the sameimage and aspect ratio as the image 420.

As shown in FIG. 4C, the manipulated and decoded video from the endpointis transferred to the wide screen display. The wide screen display,which has been set to the zoom mode, receives the manipulated decodedvideo having the aspect ratio of about 4:3. In the wide screen display,the image is then symmetrically enlarged to reach the entire width ofthe wide screen display. Since the height of the manipulated image isgreater then the height of a conventional 16:9 image, the two rectangles422 aZ and 422 bZ overflow beyond the size of the wide screen displayand are not shown on the display. The result is that only the area 411Z,which represents the original 16:9 composite image 410 of FIG. 4A, isdisplayed over the wide screen display.

FIGS. 5A-5C illustrate an example video image during three differentprocessing stages of a composite image. In this example, the image isprocessed for display over a wide screen display while the wide screenis adjusted to the wide mode. In wide mode, the video image isasymmetrically enlarged by the wide screen display so that the image canbe displayed substantially over the entire screen.

In FIG. 5A, the image 410 is the same image discussed previously in FIG.4A. Image 410 is a composite image having the aspect ratio of 16:9 atthe output of the builder 222 (FIG. 2). The composite image istransferred to output scaler 224 (FIG. 2). The output scaler 224 (FIG.2) is set to asymmetrically scale the image by increasing an originalheight H of the image by a factor of 4/3. As a result, a distorted andmanipulated image 440 shown in FIG. 5B is produced that has the aspectratio of 4:3. It can be observed that the image 440 is distorted orpulled up. For example, a circle 412 becomes an ellipse 412 d, and asquare 418 becomes a rectangle 418 d.

The manipulated (distorted) image 440 is encoded by the encoder 226(FIG. 2) and is transmitted to the endpoint (not shown). At theendpoint, the compressed video is decoded. Decoding at the endpointproduces an uncompressed video image, which is still distorted and hasthe same shape as the manipulated video 440. The distorted image is thentransferred to the wide screen display associated with the endpoint. Asshown in FIG. 5C, the wide screen display, which has been set to widemode, asymmetrically enlarges the image. To substantially cover theentire wide screen, the width of the manipulated video 440 is enlargedby a factor of 4/3. This asymmetrical enlargement corrects the distortedimage 440 and generates undistorted image 450 having the ratio of about16:9 that substantially covers the entire wide screen. It can be seenthat over the wide screen the circle 412 w and square 418 w have beencorrected and have the same shape as it was in the original compositeimage 410.

FIG. 6 is a flow chart showing steps of an exemplary method 500 forsetting a video output module 220 (FIG. 2) to prepare a compressed videostream for display on a wide screen display. The disclosed method 500can be preformed by the Management Conference System (MCS) (not shown),which is associated with the VCCU. The disclosed method 500 can be usedto set the builder 222, the output scaler 224, and/or the input scalers216 a-c of FIG. 2. The disclosed method 500 is initiated at a startscreen task 510 for each one of the endpoints associated with a videoconference during establishing the connection with the endpoint. Thetask 500 obtains the particular parameters of the endpoint.

At step 520, a decision is made whether the aspect ratio of the displayof the endpoint is known to the VCCU. The aspect ratio may be known, forexample, if during reserving the video conference one of the parametersbeing loaded or mentioned during the reservation of the video conferenceis the aspect ratio of the display being used by the endpoint. In somecases, the endpoint can implement a configuration interface, in whichthe type of the display and its mode of operation (Normal, Wide, Zoom)is declared. The endpoint can declare the information on the wide screenas part of its set of capabilities. If the aspect ratio of the displayis known, the disclosed method 500 proceeds to step 534. If the aspectratio of the endpoint is not known at step 520, then the VCCU mayrequest at step 522 that the user associated with the relevant endpointdefine the aspect ratio of the display for the endpoint. A waitingperiod T1 may be taken at step 522 for a response. The waiting period T1may be in the range of a few seconds to a few minutes. Requesting theaspect ratio may be transferred to the user by different techniques asare disclosed below.

In one exemplary technique of determining the aspect ratio of thedisplay associated with an endpoint, the disclosed method can create avideo message. The video message may be embedded with the video signaland may request the user to define the type of screen by selecting oneof displayed options. The response of the user may be done by pressingthe appropriate button on the remote control of the endpoint and sendingDTMF signals. Other exemplary techniques may use other types ofsignaling, including but not limited to FECC (ITU standard H.281). Moreinformation on video message techniques is disclosed in US PatentApplication Publication No. 20030174202, which has Ser. No. 10/346,306and has been incorporated herein by reference in its entirety.

In other exemplary embodiments of determining the aspect ratio of thedisplay associated with an endpoint, the disclosed method 500 caninitiate an Interactive Voice Response (IVR) session during step 522.The IVR session can request the user to select one of the two optionsfor defining the type of screen being used. The user may respond againby pressing the appropriate key in the remote control at the endpointand sending DTMF signals back to the VCCU. The DTMF signals can beembedded and added to the audio signals generated at the endpoint.Alternatively, the DTMF signals can be out-of-band by using knownout-of-band signaling, such as IETF RFC 2833. Other embodiments of thedisclosed method 500 may use other techniques known in the art to informthe VCCU about the type of display associated with the endpoint. Theseother techniques can include but are not limited to FECC or speechrecognition techniques.

At step 530, a decision is made whether the aspect ratio is known. Ifthe aspect ratio is known at step 520 or a response was received at step530, then the disclosed method 500 proceeds to step 534. The aspectratio may still be unknown at step 530 if the user has not respondedduring the period of T1 or if the user is not aware of the type ofscreen being used. In addition, the aspect ratio may still be unknown atstep 530 if the disclosed method 500 does not include step 522.Nevertheless, the disclosed method 500 can proceed to step 532 wheremethod 500 assumes that the aspect ratio of the display used with acurrent endpoint is 4:3.

At step 534, a layout is selected from one or more group of layouts. Thelayouts are prepared in advance by a designer. The designer can create aplurality of layouts by using similar techniques to those disclosedabove in conjunction with FIGS. 3A-3F, for example. Each layout matchescertain layout parameters. The layout parameters can include: thedisplay aspect ratio, the number of segments, and the location of thesegments in the layout, and the relation between the sizes of thedifferent segments in the layout, among other parameters. For each typeof layout, the MCS contains the setting of certain elements in the videounit 200 (FIG. 2). For example, the MCS can contain the settings of thescalers and/or the builder. A variety of layouts matching the relevantaspect ratio of the display can be selected from the group of layouts.The user can select the preferred layout by using the video messagetechniques, such as those disclosed in U.S. Patent ApplicationPublication No. 20030174202, which has Ser. No. 10/346,306 and has beenincorporated herein by reference in its entirety.

Next, a decision is made at step 540 whether the aspect ratio of thedisplay is 16:9. If the ratio is 16:9, then a decision is made at step544 whether the relevant endpoint can handle an image with an aspectratio of 16:9. An endpoint can handle an aspect ratio of 16:9 if it usescompression standards such as H.264 or H.263 with a capability of usingcustom picture format. On the other hand, the endpoint may not handlethe aspect ratio of 16:9, because the endpoint is limited and can handleonly compression algorithms such as H.261 or H.263 without custompicture format. If it is determined at step 544 that the endpoint canhandle an image having the aspect ratio of 16:9, then the disclosedmethod 500 may proceed to step 548 where the output module is setaccordingly. If it is determined at step 544 that the endpoint cannothandle the aspect ratio of 16:9, then the disclosed method 500 proceedsto step 546 where the type of the manipulation technique is selected.

At step 546, the disclosed method 500 can add one or more rectangularsegments to increase the height of a video image and change the aspectratio of 16:9 to 4:3, according to the manipulation technique disclosedabove in conjunction with FIGS. 4A-4C. Alternatively, the disclosedmethod 500 can distort the composite image by increasing an originalheight by a factor of 4/3 compared to the width, such as disclosed abovein conjunction with FIGS. 5A-5C. The builder 222 (FIG. 2) is setaccording to the selected manipulated technique. For example, if theselected manipulated technique includes adding the rectangular segments,then the builder 222 (FIG. 2) may be set to add these rectangularsegments. If the other technique is selected, then the output scaler 224(FIG. 2) may be adjusted to distort the image by increasing the heightby a factor of 4/3 compared to the width. Then, the builder 222 and theoutput scaler 224 (FIG. 2) may receive all the parameters associatedwith the decoded streams that compose the composite image. Among others,the parameters may include information for receiving the decoded streamsfrom DVCI 230 (FIG. 2) and the location in the layout in which each oneof the decoded video has to be placed in order to create the composite16:9 layout. If the display mode is known, (wide or zoom) then themodification technique may be selected accordingly.

After setting the builder and/or the scaler, a message can be sent tothe endpoint. The message may be embedded in the video or can be issuedas an audio message, informing the user to set the wide screen displayto the appropriate mode. For example, if the distortion technique hasbeen selected, then the user is instructed to set the wide screendisplay to the wide mode. However, if the technique of addingrectangular segments has been selected, then the user is instructed toset the wide screen display to the zoom mode. After instructing the userto set the wide screen display, the disclosed method 500 is terminatedat step 550 for this endpoint. If needed, the disclosed method 500 canbe restarted at step 510 for the next member of the video conference.

The step 548 of setting the output module can be used in one of twocases. In a first case, step 548 is initiated if it is determined atstep 540 that the display has a common aspect ratio of 4:3. In thisfirst case, there is no need for a special modification of the decodedvideo to prepare it to be displayed. Therefore, the output module 220(FIG. 2) can be set accordingly. In a second case, step 548 is initiatedif it is determined at step 544 that the endpoint can handle imageshaving the aspect ratio of 16:9. In this second case, there is no needfor a special modification of the decoded video in the output module 220to prepare it to be displayed on a wide screen. Therefore, in bothcases, the disclosed method 500 proceed to step 548 where the outputmodule 220 (FIG. 2) is set according to the appropriate standard forcomposing and encoding the composite image and transmitting it to theendpoint. After setting the output module according to the appropriatestandard, the disclosed method 500 is terminated at step 550.

In this application the words “unit” and “module” may be usedinterchangeably. Anything designated as a unit or module may be astand-alone unit or a specialized module. A unit or module may bemodular or may have modular aspects allowing it to be easily removed andreplaced with another similar unit or module. Each unit or module mayinclude any one or combination of software, hardware, or firmware.

Those skilled in the art will appreciate that the disclosed system andmethod can be implemented in the form of additional software residing inthe VCCU that performs the techniques disclosed in the presentapplication. In addition, those skilled in the art will appreciate thatthe disclosed system and method can be implemented in the form ofadditional hardware added to the VCCU or distributed among the VCCU andthe endpoints that performs the techniques disclosed in the presentapplication.

Furthermore, those skilled in the art will appreciate that the disclosedsystem and method can be used with a variety of compression standards,including but not limited to: H.264, H.263, H.261, MPEG 1, MPEG 2, andMPEG 4. More information on these standards may be obtained from theInternational Telecommunications Union (ITU) website atURL://http://www.itu.int/ or from the website atURL://http://www.mpeg.org/.

In the description and claims of the present disclosure, each of theverbs, “comprise,” “include,” and “have,” and conjugates thereof, areused to indicate that the object or objects of the verb are notnecessarily a complete listing of members, components, elements, orparts of their respective subjects or verb.

The foregoing description of preferred and other embodiments is notintended to limit or restrict the scope or applicability of theinventive concepts conceived of by the Applicant. In exchange fordisclosing the inventive concepts contained herein, the Applicantdesires all patent rights afforded by the appended claims. Therefore, itis intended that the present invention include all modifications andalterations to the full extent that they come within the scope of thefollowing claims or the equivalents thereof.

1. A method of composing a video layout, comprising: receiving at an MCUa plurality of compressed video streams from a plurality of endpoints;decoding of the compressed video streams; scaling of the decoded videostreams; selecting from the decoded and scaled video streams a pluralityof scaled video segments, at least one of the scaled video segmentshaving an aspect ratio of about 4:3; and composing an integer number ofthe selected scaled video segments in a video layout having an aspectratio of about 16:9, preserving the aspect ratio of the selected scaledvideo segments.
 2. The method of claim 1, wherein each of the scaledselected scaled video segments has an aspect ratio of about 4:3.
 3. Themethod of claim 2, wherein the video layout comprises twelve scaledvideo segments arranged in a 4×3 matrix.
 4. The method of claim 2,wherein the video layout comprises two large scaled video segments andfour small scaled video segments and wherein the area of each of thelarge scaled video segments is about four times the area of each of thesmall scaled video segments.
 5. The method of claim 2, wherein the videolayout comprises one large scaled video segment and eight small scaledvideo segments and wherein the area of the large scaled video segment isabout four times the area of each of the small scaled video segments. 6.The method of claim 2, wherein the video layout comprises one largescaled video segment and three small scaled video segments and whereinthe area of each of the large scaled video segments is about nine timesthe area of each of the small scaled video segments.
 7. The method ofclaim 1, further comprising transmitting the video layout to anendpoint.
 8. The method of claim 7, wherein the endpoint comprises a16:9 display and a codec that is not capable of processing a 16:9 videoimage.
 9. The method of claim 8, further comprising, prior totransmitting the video layout to the endpoint, converting the videolayout from 16:9 to 4:3.
 10. The method of claim 9, wherein convertingthe video layout from 16:9 to 4:3 comprises adding a rectangular segmentof height to the layout, wherein the rectangular segment of height isfilled with a background color.
 11. The method of claim 10, wherein onehalf of the added rectangular segment of height is added above the videolayout and one half of the added rectangular segment of height is addedbelow the video layout.
 12. The method of claim 9, wherein convertingthe video layout from 16:9 to 4:3 comprises distorting an originalheight of the 16:9 video image by a factor of about 4/3.
 13. The methodof claim 1, further comprising transmitting an instruction to theendpoint to set the 16:9 display to a zoom mode.
 14. A method ofcomposing a video layout, comprising: receiving at an MCU a plurality ofcompressed video streams from a plurality of endpoints; decoding of thecompressed video streams; scaling of the decoded video streams;selecting from the decoded and scaled video streams a plurality ofscaled video segments, at least one of the scaled video segments havingan aspect ratio of about 4:3; and composing an integer number of theselected scaled video segments in a video layout having an aspect ratioof about 16:9; transmitting the video layout to an endpoint comprising a16:9 display and a codec that is not capable of processing a 16:9 videoimage; and converting the video layout from 16:9 to 4:3, comprising:adding a rectangular segment of height to the layout, wherein therectangular segment of height is filled with a background color, priorto transmitting the video layout to the endpoint.
 15. The method ofclaim 14, wherein the act of adding a rectangular segment of height tothe layout, wherein the rectangular segment of height is filled with abackground color, prior to transmitting the video layout to the endpointcomprises: adding one half of the rectangular segment of height abovethe video layout; and adding one half of the rectangular segment ofheight below the video layout.
 16. A method of composing a video layout,comprising: receiving at an MCU a plurality of compressed video streamsfrom a plurality of endpoints; decoding of the compressed video streams;scaling of the decoded video streams; selecting from the decoded andscaled video streams a plurality of scaled video segments, each of thescaled video segments having an aspect ratio of about 4:3; and composingan integer number of the selected scaled video segments in a videolayout having an aspect ratio of about 16:9, comprising one or morelarge scaled video segments and a plurality of small scaled videosegments, wherein the area of each of the large scaled video segments isapproximately an integral multiplier of the area of each of the smallscaled video segments.
 17. The method of claim 16, wherein the videolayout comprises two large scaled video segments and four small scaledvideo segments and wherein the area of each of the large scaled videosegments is about four times the area of each of the small scaled videosegments.
 18. The method of claim 16, wherein the video layout comprisesone large scaled video segment and eight small scaled video segments andwherein the area of the large scaled video segment is about four timesthe area of each of the small scaled video segments.
 19. The method ofclaim 16, wherein the video layout comprises one large scaled videosegment and three small scaled video segments and wherein the area ofeach of the large scaled video segments is about nine times the area ofeach of the small scaled video segments.
 20. The method of claim 16,further comprising: converting the video layout from 16:9 to 4:3; andtransmitting the converted video layout to an endpoint comprising a 16:9display and a codec that is not capable of processing a 16:9 videoimage.